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ABSTRACT 

In this paper a fusion technique is presented for the 
3-D modeling of the free navigation space. The fusion 
technique could be used by an autonomous robot to 
model and acquire the navigation space and for the 
extraction of the 3-D map from the environment. The 
fusion technique is based on the appropriate synthesis of 
two different sensory data generated by a vision camera 
and a laser scanner. 
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1. INTRODUCTION 

Humans and animals are born with multiple 
senses that allow them to develop a very clear picture of 
our world. Thus, in the design of autonomous robots, or 
robust vision systems, the cooperation of the multiple 
sensors is needed to enhance the system's chances of 
success in a complex environment. Of particular 
importance is how the robot formulates a 3-D image of 
the environment. We intend on using the fusion of a 
color segmented picture and a laser range finder to 
develop a working model of the space. 

The idea of fusing two sensory signals to 
formulate a 3-D image is by far not new. Extensive 
work has been done in this field of machine vision. We 
encourage the reader to further enhance their 
background by looking at [11 and [2]. 

Several very interesting approaches have been 
done in the field of fusing intensity and range data. Of 
particular interest is the work done by J.K Aggarwal et. 
al. [3]. 

In particular in [3J, a method of fusion between 
range data and intensity is presented. One major goal of 
this approach was to minimize the extensive amount of 
time required in sensing the range data. Using potential 
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points of interest from the intensity data as directors for 
the range sensor, the range sensor senses those points of 
interest and then the range data and intensity data are 
combined to form the graph. 

Another method is discussed in [4]. This method 
uses the most dominant sensor at one time (range or 
intensity) for seed segmentation. This segmented area 
then again uses the data from the range and intensity 
for region merging. The authors used this method to 
minimize the lighting difficulties and color variance 
(intensity image segmentation), and to minimize the 
difficulties range image segmentation has with sloped 
surfaces. 

Other related work is available in [5], [6], and [7]. 

It is interesting to be mentioned that the laser 
scanner has very high resolution (similar to a digital 
image) very close to the scene surface. This means that 
in longer distances, such as 1 m or longer than that, the 
resolution changes drastically. For instance, in a distance 
of 1 m, the size of the laser spot has diameter of I cm 
approximately. Thus, how the image 3-D model will be 
developed under these conditions. 

We propose a methodology, which hopefully will 
correct some of these problems, such as long distance 3- 
D image modeling and color difficulties, by using the 
intensity data as the director for the laser. More 
specifically, we use a fuzzy image segmentation 
technique [9] for the sufficient separation of the image 
regions with different color. In addition, we generate an 
image with a low resolution equivalent to the laser one, 
thus the fusion of these sensory data to be possible and 
the final 3-D image to contain less "noise" on the 
scanned surfaces. 

This paper is organized into five sections. Section 2 
discusses the laser range and some of the existing 
problems. Section 3 presents briefly the fuzzy image 
segmentation technique for the separation of the regions. 
Section 4 describes the fusion methodology of these two 
different sensory data. The last section summarizes the 
overall presentation. 
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2. LASER RANGE DATA ACQUISITION 


Ideally we would like to scan the picture area with 
a laser scanner. This would provide us with a range 
map that could be "fused " with our intensity segmented 
map. However the inhibitive high cost of laser scanners 
is not within our university’s budget. 

That leaves us with two options; we can attempt to build 
our own laser scanner or we can purchase a point by 
point laser range detector that can be manipulated in a 
xy plane. Due to the time constraints of this paper we 
will assume that we have the funds to purchase a point 
by point detector. It is also worth noting that it would 
probably take us a considerable amount of time to build 
the complex circuitry required for a laser scanner. Our 
objective is to develop a new fusion technique. 

One company that has a variety of laser distance 
meters is RIEGL[8]. Features included in these 
detectors are RS233 interface (as long as measure times 
are > 50 ms), immunity to electro-magnetic and acoustic 
interference, high speed and accuracy, and a starting 
beam width of s 1cm with a divergence of 3.2 mrad. 
Since our robot is designed primarily for enclosed 
hazardous environments the laser sensor range does not 
need to be exceptionally long. The meter fulfilling our 
needs the best is the LD90-210-CX/C6. It’s sensing 
range is from 1 meter to 15 meters. In addition [8] also 
has a biaxial rotary mount (BD 90). The mount enables 
us to selectively pick the intensity segmented areas. 

Theoretically a beam width the size of a pixel 
would be ideal. This would permit a one to one fusion 
with the pixels form the intensity segmentation. 
However technology has not yet reached this minute 
beam width. The authors of this paper are aware of 
very small diameter beam widths but these lasers are 
not yet available. 

After the separation of the intensity scan into 
specific colors the area of each color segment is 
determined and put in xy coordinates. Once an area is 
defined the bidirectional mount is manipulated so that 
the laser range meter can determine the range of the 
area in question. When the scanning of the area is 
complete another color area is scanned until the entire 
intensity frame has been scanned. The intensity map 
and the range map are then fused to form our first 
attempt at the 3-D image. 

One of the critical features of the laser scanner is 
the non-uniform scanning of the scene. More specifically, 
since the mechanical XY movement mechanism used by 
the laser scanner is imperfect, there is the possibility 
that the laser beam to overlap two consecutive scanned 
point, or to be far away from each other, as shown in 
figure 1. A solution to this particular problem is the 
scanning of the same scene area twice, and the 
averaging of two distance values by reducing the 
possible error. 



Figure 1: The imperfect scanning 


3. FUZZY IMAGE SEGMENTATION 

Segmentation and edge detection in color images 
are not deeply investigated in the literature. Some 
methods separate the three color components and work 
on each independently. Three different histograms are 
drawn and after segmentation or edge detection, the 
results are combined to form segmented or edge 
detected color images. The results using these techniques 
are false because of three reasons: one, segmentation or 
edge detection based on color alone is not complete. The 
relative position of the pixels and their color in their 
neighborhood are important as well. Two, in real images 
there is no sharp valley in histograms. Three, once one 
pixel color is mapped to three locations on three 
histograms, pixel color information is scattered. And 
once this is done for thousands of pixels, all available 
information is scattered and mixed. After thresholding 
doing the reverse and putting the three components 
back together is not an easy job. This technique is 
usually used for images having a limited number of 
segments or clusters. 

The technique described in this section is 
applicable to both color and gray level images having an 
unlimited number of clusters. A cluster is defined as a 
collection of touching pixels which have almost the same 
color or the change in color is gradual. Since an 
unlimited number of clusters could be detected from an 
image, displaying each cluster in a different image 
would not be practical. What we have done is displaying 
all clusters in one image. Each cluster is shown using a 
color that reflects the main color of the cluster. 

Before segmentation can begin a rough edge 
detection must be performed. The used edge detection 
technique is not described here but it is sensitive enough 
to detect all sorts of edges (crisp, fuzzy, thin, and week). 
A histogram table is also generated and sorted according 
to the number of pixels. The first three entries of the 
table for a particular image are shown below: 


21 27 23 892 
21 26 23 640 
8 12 28 591 

The numbers represent red, green, blue, and the 
number of pixels in the image having that particular 
color vector, respectively. Segmentation is performed in 
the following steps: 1- find big clusters, 2- expand 
clusters, 3- find medium size clusters, 4- expand clusters, 
5-find small clusters, 6- fill in the blanks. 


where R, G, and B are the three color components. 
Variance membership function is shown below: 




0 if Variance < 

1 if Variance u > a 2 
Vananc^ -a, 

<h~<h 


Big clusters are defined as those having a 
minimum of 150 pixel members. To find big clusters 
instead of looking for valleys in the histogram we look 
for peaks. The first peak is the first entry in the 
histogram table. Therefore, the seed color is known and 
the seed location must be found. The image is scanned 
for that particular color vector and once found it is 
test-grown and pixels marked until a detected edge is 
reached and the size of the test-grown cluster is noted, 
t his is done for all unmarked pixels having the seed 
color. The seed location that uses the most of the seed 
color pixels and is big is chosen as the seed location and 
its color as the cluster color. The seed is then grown. 
While the seed is growing and pixels are added to the 
cluster, each pixel is marked and its entry in the 
histogram table decrement. Meaning, that particular 
pixel is not available any more and does not contribute 
to the histogram table. When the seed is all grown, the 
histogram table is resorted and the first entry, now, is 
the second peak of the histogram. This growing process 
is repeated until there are no more big clusters left. 

Since the edge detection algorithm used is very 
sensitive to variance, the detected edges are thick 
especially in the case of fuzzy edges. Therefore, big 
clusters must be expanded using fuzzy logic to fill out 
that gap before smaller clusters are found. Big cluster 
expansion is performed in two steps: In the first step 
each cluster is expanded (surrounding pixels added to 
the cluster) based on two rules: 

1. The absolute variance (difference between the 
pixel's color and the seed/cluster's color) is low. and 

2. The relative variance (difference between the next and 
the previous pixel's color) is low. This is used to 
include gradual changing shade areas as well. 

In the second step each cluster is expanded based on the 
following rule: 

For each pixel the absolute variance with the 
surrounding clusters and the distance (in number of 
pixels) with that cluster is evaluated and their product 
recorded (degree of farness). If the lowest product value 
(which is for the closest cluster in terms of color and 
distance) is lower than a threshold, the pixel is added to 
the cluster. 

The following equation is used to calculate 
variance between pixel 1 and 2. 


Variance = (/^ - /?, f + (G 2 - G x ) 2 + ( B 2 - B x ) 2 



Averaging is used in anding the two fuzzy membership 
functions. 

Medium size clusters are found the same way big 
ones were but this time they are grown based on the 
first step of expansion rules (minimum size 30 pixels 
total). They are then expanded based on the second step 
of expansion rules. 

To find small clusters, for each undecided pixel 
in the image the degree of farness with the nearby 
clusters is evaluated. 

If the lowest value is higher than a threshold, use 
the pixel as a seed and test-grow it based on the first 
step of expansion rule. If the test-grown cluster has at 
least 6 pixel members and has a big contrast with the 
closest cluster (its existence is of importance), grow it, 
otherwise leave it undecided. To fill in the blank for all 
undecided pixels the closest cluster (in distance and 
color) is found and the pixel is added to that cluster. 
Figure 2 shows a colored image and its segmentation. 
Figure 3 shows a grey image followed by its segmented 
one. 


4. THE FUSION METHODOLOGY 
4.1. Methodology 

The fusion methodology presented here is based 
on the synthesis of different sensory data generated by 
a vision system and a laser scanner. In particular, the 
camera receives an image from the environment. The 
digital form of the image P(ij), l<ij<n, where "n" 
represents the size of the image matrix, is stored in the 
main memory. The digital image represents a 2-D array 
of gray level values or color values (gv(ij)) for the image 
pixels. At the time that the image is grabbed by the 
camera, the laser scanner "scans" the same area of the 
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Figure 2: a) A colored image 

Figure 3: a) 

A grey level image 

b) Its segmentation 

b) 

Its segmentation 







environment by producing an 2-D array D(k,m). The 
values d(k,m) of the D array represent the distances of 
the laser scanner from the surface of the current 
navigation space at the particular area from which the 
image was grabbed. The D array is also stored in the 
memory. From this point, the fusion technique takes 
place. More specifically, a fuzzy segmentation technique 
is applied on the image values to define the regions with 
"color" similarities. An edge detection technique is also 
applied on the same image to define and separate the 
boudaries of the image regions [9]. Since the resolution 
of the P image-matrix is higher to the D distance-matrix, 
the resolution ratio (R(F)= [R(P)/R(D)D is calculated. 
This R(F) ratio will guide the image pyramid generation 
technique for the selection of the pyramidal level of the 
image (PL) with the closest resolution to the D matrix, 
figure 4. 



At this point, the fusion matrix F is generated by 
combining the coordinates of these two matrices PL and 
D respectively. More specifically, the elements of the F 
matrix have four values: the x' and y' coordinates, the 
d distance of these x'y' coordinates, and the gv grey 
level value which cooresponds to x'y'; f(x\y',d,gv). The 
3-D image model (F3) for the original image will be 
generated by the F matrix with the expansion of each 
f(x'y\d,gv) element by R(F) pixels. More specifically, 
f(*\y\d,gv) — > S{ P } q 

where S represents the set of 4 neighbor pixels, so 
that R(F)% 4^ . Thus, 4^ neighbor pixels of the 3-D 
image model will share the same distance d; 
p(i+wj+z,d,gv), where w,ze{l,2,...2^ }. 

At this point, the colored image regions separated by the 
segmentation method will be used for "smoothing" the 
possible unevenness of the 3-D model. More specifically, 
there will be cases where the pixels, which belong to 
different colored regions, share the same distance d. In 
this case, the distance d(Rc) which represents the 
majority of the pixels in the region Rc, with color "c", 
will be used to redefine the distance of those pixels, 
which share a distance different than d(Rc). 


Figure 5: A grey level image 



Figure 6: The reduced image using pyramidal 
procedure . 


4.2. Illustrative Example 

In this subsection we present graphically the fusion 
methodology by using a simple grey level image, figure 

5, the reduced image using the pyramidal process, figure 

6, a generated distance matrix with equivalent 
resolution, figure 7, the fused 3-D image at the reduced 
size, figure 8, and the 3-D model of the original image, 
figure 9. 



^—7 

Figure 4: The pyramidal reduction of 
an image 



Figure 7: The distance matrix (in a 3-D form) 
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5. CONCLUSIONS 

In this paper a fusion methodology has been 
presented. This methodology combines two different 
type of sensory data for the generation of a 3-D image 
model. The sensory data used by this method were the 
distances produced by a laser scanner and the pixels 
average intensities (or color) of a 2-D digital image 
grabbed by a vision camera. The problem of the 
different resolutions for these two sensory data was 
solved by the reduction of the image resolution, the 
fusion of the different data and the reconstruction of the 
original image by using a fuzzy image segmentation 
technique. Since the laser scanner was available for this 
experimental work, only feasibility simulated result# 
were presented. 


